
AFFECTIVE CONTROL OF INFORMATION SYSTEMS 
BACKGROUND OF THE INVENTION 

5 Field of the Invention. 

This invention relates generally to control of data processing or information systems, and 
in particular to control of such systems based in part upon the detected apparent affective or 
emotional state of a user. 

10 Description of the Related Art 

In the history of human interaction with information systems, system reaction to the user 
actively engaged with the system generally has been based upon programmatic response to a 
Q user's intentional communicative acts, such as keyboard entry, mouse manipulation, voice input 
fU and the like. While certain information systems and applications respond to forms of 
information from a user that are not intentional communicative acts by the user, such as 
autonomic conditions in medical monitoring systems or biometric parameters for user 
:= identification in security systems, these system responses are generally not within the context of 

i : 

n active engagement of the user with the system. 

Yet for humans actively engaged with each other, an individual's response to another 
. [Jo involves considerably more than reacting to the mere intentional communicative acts of the 
parties. In particular, effective interaction with others requires perceiving and reacting to the 
apparent affective or emotional states of others as well as to their intentional communicative acts. 

An information system capable of determining a user's apparent affective state would be 
capable of altering its behavior in response to that apparent state, resulting in more effective 
25 interaction with the user. For example, in applications such as email, when a user is in 
communication with other users, a user's rash emotional reaction in communication may result 
in the user's transmitting an undesirable message. The system in this example, then, having 
determined that the user's apparent emotional state is such that the user's communication may be 
a rash response, may prompt the user for confirmation prior to transmission. What is needed for 
30 more effective human interaction with information systems is for information systems to 
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determine and react appropriately to the apparent affective or emotional state of humans with 
whom they are actively engaged. 

In addition to human-computer interaction, humans also use information systems as 
intermediaries in communication with other humans, as in email and word processing 
5 applications. Because effective interaction with other humans requires conveying affective or 
emotional states, a number of means of conveying such states have been developed by 
convention for written communications. For example, a correspondent's typing in all capital 
letters may serve to convey to the recipient the same emotional state that yelling conveys in 
spoken communications. Additionally or in the alternative, special combinations of common 
10 characters may be typed by a correspondent to convey an emotional state to a recipient, such as 
the now familiar colon-right parenthesis combination :) used to convey happiness or humor. 
» Acronyms typed by a correspondent and understood in common with the recipient can serve the 
3 same function, for example "LOL" (laughing out loud) and "ROTF" (rolling on the floor), 
y Depending upon display capabilities of the information systems of correspondent and recipient, 
15 the correspondent may convey emotional or affective state to the recipient by typing keystrokes 
LJ or using function keys to insert special characters (such as a smiley face © ), to change a font to 
bold or italic or to underline for emphasis, or to color font in appropriate portions of text. To 
7i convey affective or emotional state in any of these and like conventions used in written 
^ communications, the correspondent must intentionally direct an input device, such as a keyboard 
io or a mouse, to insert characters or mark text to convey the correspondent's emotional state. 
What is needed is for the correspondent's information system, without any intentional direction 
from the correspondent, to determine the apparent affective or emotional state of the 
correspondent and react by marking appropriate text in such a way that by convention conveys 
the emotional state of the correspondent to the recipient. 
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SUMMARY OF THE INVENTION 



The present invention shows a way for information systems to determine the apparent 
emotional or affective state of a user engaged with the system and, further, for information 

5 systems, based upon such determination, to react accordingly. 

It is an object of the present invention to provide means for information systems to 
determine the apparent emotional or affective state of a user. It is a further object of the present 
invention to enable information systems to make intelligent decisions in applications programs 
based upon a determined apparent emotional or affective state of a user. It is a further object of 

10 this invention, having determined the apparent emotional or affective state of a correspondent, to 
convey that apparent state in communications transmitted by the correspondent to one or more 

3 recipients. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



Other objects, advantages, features and characteristics of the present invention, as well as 
methods, operation and function of related elements of structure, and the combination of parts 
and economies of manufacture, will become apparent upon consideration of the following 
description and claims with reference to the accompanying drawings, all of which form a part of 
this specification, wherein like reference numerals designate corresponding parts in the various 
figures, and wherein: 

FIG. 1 is a block diagram of a general purpose information handling system, which may 
be used to implement the invention. 

FIG. 2 is a flow chart of a process for applying a determination of apparent affective state 
to marking of text in a text composition application. 

FIG. 3 is a flow chart of a process for applying a determination of apparent affective state 
in an interpersonal assistant application that warns a user of possible inadvisability of sending 
email. 

FIG. 4 is an exemplary warning output of the process shown in FIG. 3. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

In the following detailed description of the preferred embodiments, reference is made to 
5 the accompanying drawings which form a part hereof, and in which is shown by way of 
illustration specific preferred embodiments in which the invention may be practiced. These 
embodiments are described in sufficient detail to enable those skilled in the art to practice the 
invention, and it is to be understood that other embodiments may be utilized and that changes 
may be made without departing from the spirit or scope of the invention. To avoid detail not 
10 necessary to enable those skilled in the art to practice the invention, the description may omit 
certain information known to those skilled in the art. The following description is, therefore, not 
to be taken in a limiting sense, and the scope of the invention is defined only by the appended 
claims. 

ry FIG. 1 shows a block diagram of an information handling system 100 in accordance with 

r'ys the present invention. In this embodiment, processor 102, system controller 112, cache 1 14, and 
^^'1 data-path chip 1 18 are each coupled to host bus 1 10. Processor 102 is a microprocessor such as a 
^= 486-type chip, a Pentium®, Pentium II®, Pentium III®, or the like suitable microprocessor. 
LJ Cache 1 14 provides high-speed local-memory data (in one embodiment, for example, 512 KB of 
n data) for processor 102, and is controlled by system controller 1 12, which loads cache 1 14 with 
^l|o data that is expected to be used soon after the data is placed in cache 1 12 (i.e. in the near future). 
Main memory 1 16 is coupled between system controller 1 12 and data-path chip 118, and in one 
embodiment, provides random-access memory of between 16 MB and 128 MB of data. 

In one embodiment, main memory 116 is provided on SIMMs (Single In-line Memory 
Modules), while in another embodiment, main memory 1 16 is provided on DIMMs (Dual In-line 
25 Memory Modules), each of which plugs into suitable sockets provided on a motherboard holding 
these components and many of the other components shown in FIG. 1. Main memory 116 
includes standard DRAM (Dynamic Random-Access Memory), EDO (Extended Data Out) 
DRAM, SDRAM (Synchronous DRAM), or the like suitable memory technology. System 
controller 1 12 controls PCI (Peripheral Component Interconnect) bus 120, a local bus for system 
30 100 that provides a high-speed data path between processor 102 and various peripheral devices. 
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such as video, disk, network, etc. Data-path chip 1 18 is also controlled by system controller 112 
to assist in routing data between main memory 1 16, host bus 110, and PCI bus 120. 

In one embodiment, PCI bus 120 provides a 32-bit-wide data path that runs at 33 MHz. 
In another embodiment, PCI bus 120 provides a 64-bit-wide data path that runs at 33 MHz. In yet 
5 other embodiments, PCI bus 120 provides 32-bit-wide or 64-bit-wide data paths that run at 
higher speeds. In one embodiment, PCI bus 120 provides connectivity to I/O bridge 122, 
graphics controller 127, and one or more PCI connectors 121, each of which accepts a standard 
PCI card (not shown). In one embodiment, I/O bridge 122 and graphics controller 127 are each 
integrated on the motherboard along with system controller 112, in order to avoid a board-to- 
10 connector-to-board signal crossing interface, thereby providing better speed and reliability. In 
the embodiment shown, graphics controller 127 is coupled to a video memory 128 that includes 
..^ memory such as DRAM, EDO DRAM, SDRAM, or VRAM (Video Random- Access Memory), 
'^3 and drives VGA (Video Graphics Adapter) port 129. VGA port 129 can connect to VGA-type or 
ry SVGA (Super VGA)-type displays or the like. Other input/output (I/O) cards having a PCI 
pffe interface can be plugged into PCI connectors 121 . 

: id' 



In one embodiment, I/O bridge 122 is a chip that provides connection and control to one 
or more independent IDE connectors 124, to one or more SCSI connectors 125, to one or more 
USB (Universal Serial Bus) ports 126, and to an ISA (Industry Standard Architecture) bus 130. 
In this embodiment, IDE connector 124 provides connectivity for up to two or more standard 



Czo IDE-type devices, in particular those for non-volatile memory storage ano/or retrieval such as 
hard disk drives, CD-ROM (Compact Disk-Read-Only Memory) drives, DVD (Digital Video 
Disk or Digital Versatile Disk) drives, or TBU (Tape-Backup Unit) devices. As will be 
appreciated by those skilled in the art, client systems in a network, such as web pads, need not be 
equipped with any such non-volatile memory storage devices, relying instead upon the function 

25 of such devices in a server to which the client is connected. 

In one similar embodiment, two IDE connectors 124 are provided, each providing an 
EIDE (Enhanced IDE) compliant architecture. In the embodiment shown. Small Computer 
System Interface (SCSI) connector 125 provides connectivity for preferably up to seven or 
fifteen SCSI-type devices depending on the version of SCSI supported by the respective 

30 embodiment. In one embodiment, I/O bridge 122 provides ISA bus 130 having one or more ISA 
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connectors 131 (in one embodiment, three connectors are provided). In one embodiment, ISA 
bus 130 is coupled to I/O controller 152, which in turn provides connections to two serial ports 
154 and 155, parallel port 156, and FDD (Floppy-Disk Drive) connector 157. In one 
embodiment, FDD connector 157 is connected to FDD 158 that receives removable media 
5 (floppy diskette) 159 on which data and/or program code 160 is stored. 

In one such embodiment, program code 160 includes code that controls programmable 
system 100 to perform an application program as described in accordance with the invention. In 
an embodiment typical for client systems, and characteristic of "thin clients" such as web pads, 
serial port 1 54 is connectable to a computer network such as a local network or the Internet, and 
10 such network has program code 160 that controls programmable system 100 to act as a client, 
receiving and interpreting data sent by a matching server computer application. In another such 
embodiment characteristic of server systems, serial port 154 is connectable to a computer 
';i network such as a local network or the Internet, and special program code 160 within 
ry programmable system 100 executes that causes programmable system 100 to act as a server, 
providing data and applications over the network to a matching client computer program that is 
capable of properly interpreting that data and applications. 
- In one embodiment, ISA bus 130 is connected to buffer 132, which is connected to X bus 

LJ 140, which provides connections to real-time clock 142, keyboard/mouse controller 144 and 
n keyboard BIOS ROM (Basic Input/Output System Read-Only Memory) 145, and to system 
:io BIOS ROM 146. FIG. 1 shows one exemplary embodiment of the present invention, however 
other bus structures and memory arrangements are specifically contemplated. It should be 
appreciated that modification or reconfiguration of information handling system 100 of FIG. 1 by 
one having ordinary skill in the art would not depart from the scope or the spirit of the present 
invention. 

25 An information system may determine the apparent affective or emotional state of a user 

based upon a variety of sources of information, which may be used singly or in combination for 
the information system to make its determination. See generally Picard, Rosalind W. 'Teaching 
Computers to Recognize and Express Emotion" in: Stork, David G. (ed.), Hal's Legacy: 2001 's 
Computer as Dream and Reality (Cambridge, MA. MIT Press. 1997), Chapter 13, Section 4. 
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In applications where the user is inputting textual content, such as word processing or 
email composition, the actual content of the input may give a good indication of the user's 
affective state. For example, the incidence of vulgar language or derogatory epithets in text may 
indicate anger or frustration, with frequency of such words providing an indication of the 
5 intensity of the user's emotional state. By monitoring the incidence and frequency of words 
indicating a user's emotional state, an information system may make a determination of the 
user's apparent affective state. 

As previously discussed in reference to the prior art, the user may mark textual input in 
such a way as to indicate the user's emotional or affective state with respect to the text or context 
10 in question. Such marking may be by way of highlighting the text in some way, such as 
alteration of font appearance by color, underlining, bold, italic, capitals, etc. In the alternative, 
n such marking may be by way of special character combinations or special characters inserted in 
■ ^ appropriate textual locations. In any case, the correspondent conveys his emotional state to the 
'.J recipient by such marking on the assumption that the recipient shares the convention that 
fig associates that marking with the affective state of the correspondent. An information system 

i.i I 

may determine a correspondent's apparent emotional state by monitoring text input for markings 
that, by convention, convey emotion. 
Ly The facts surrounding the user's composition and transmission of text in such 

^ j applications may be indicative of the user's affective state. For example, if the period of time 
^io between a user's first access of an email message and the user's reply to that message is very 
short, it is possible that the user has reacted emotionally to the received message and has 
responded rashly based upon that emotional response. Sirnilarly, a rash response may be 
indicated if the length of time used to compose the message is very short but the recipient list of 
the message is very long or the recipients include key individuals such as the head of the sender's 
25 organization. An information system may, in such circumstances, make a determination of the 
user's apparent emotional state in the composition of email by monitoring the time to reply, the 
time to compose, and the distribution list of the email. 

A user's input characteristics may also be indicative of the user's emotional state. For 
applications using standard manual input devices, pronounced force of keystrokes on a keyboard 
30 or mouse clicks on a mouse may indicate anger or frustration. Input devices such as keyboards 
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and computer mice that are capable of detecting and reporting the degree of force in user input 
are well known in the art (see, for example, U.S. Patent 5,579,238 to Krugman, U.S. Patent 
5,982,357 to Burgett and U.S. Patent 5,995,026 to Sellers). By employing such input devices, an 
information system may determine apparent user affective state by monitoring the force 
5 exercised by the user in input operations. 

A user's input characteristics may also be determined in applications using speech input 
by analysis of the user's speech. As set forth in the HaVs Legacy reference, vocal intonation is a 
common way to communicate strong feelings. Several features of speech are modulated by 
emotion; we divide them into such categories as voice quaHty (e.g., breathy or resonant, 
10 depending on individual vocal tract and breathing), timing of utterance (e.g., speaking faster for 
fear, slower for disgust), and utterance pitch contour (e.g., showing greater frequency range and 
abruptness for anger, a smaller range and downward contours for sadness). Analysis of these 
M characteristics in a user's speech by an information system provides the information system with 
fy an indication of the user's emotional state. 

pijs Autonomic responses are also indicative of a user's emotional state. While some 

•=Lj autonomic responses may be difficult to monitor unobtrusively, galvanic skin response of a 
;i user's hands or fingers may be monitored by input devices such as keyboards or computer mice, 
j J Electrodes on such input devices at appropriate points of contact with the user's hands may be 
n used by an operatively connected information system to monitor the user's skin conductivity. As 
l-5o is well known in the art (see, for example, Kasa et al., U.S. Patent 4,625,732), increased skin 
conductivity is associated with a user's affective state. Monitoring the user's skin conductivity 
thereby provides the information system with an indication of the user's apparent emotional 
state. 



25 the user's pupils and the rate at which the user blinks his eyes. It is well known in the art that 
pupil dilation and eye blink rate, just as skin conductivity, are associated with a person's 
affective state (see, for example, U.S. Patent 4,931,865 to Scarampi). By means of an 
operatively coupled video camera, an information system may monitor a user's pupil dilation 
and/or blink rate and thereby make a determination of a user's apparent emotional state. 



Other autonomic responses that may be measured fairly unobtrusively are the dilation of 
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Face recognition software technology may also be employed by this invention. Typically 
analyzing a computer graphic of a face, such software, used in conjunction with a video camera 
operatively coupled with the information system, can determine the identity of the user and the 
user's apparent emotional state. At present, several algorithmic approaches to face recognition 
5 provide sufficiently accurate performance in face recognition for application in this invention. 
Some approaches are based upon eigenvectors generated from the autocorrelation matrix of a 
computer image of a face. These eigenvectors, termed "eigenfaces" by persons skilled in the art, 
may be analyzed by vector transform followed by discriminative modeling to yield reliable face 
recognition determinations. In such approaches, some algorithms employ a linear discriminant, 
10 while others, such as that used in the Viisage Gallery application of Viisage Technology, Inc. of 
Littleton, Massachusetts, employ a quadratic discriminant. In the alternative, within these 
approaches using eigenfaces, some algorithms, such as those used in Facelt® software by 
'fi Visionics Corp. of Jersey City, New Jersey, may employ a sparse transform of the eigenfaces, 
i'y followed by a discriminative neural network analysis. 

Illustrating yet another approach, TruePace software from Miros, Inc. of Wellesley, 
'^L: Massachusetts, relies upon a method that extracts and analyzes local features from a variety of 
ii faces. Local features of a face are extracted from an eigenface by a determination of which 
j J features of a face differ most from features of other faces in a database. By applying neural 
■/] network technology to the analysis of extracted local features in a database derived from 
'■So computer images, this software improves its face recognition success rate with experience. 



expression and thereby determine the apparent emotional or affective state of the user. 

Turning now to the remaining drawings, FIG. 2 is a flow chart of a procedure for 

25 applying a determination of apparent affective state to marking of text in a keyboard text input 
and composition application. This procedure, which processes text input prior to formatted 
output, commences at 201. In 202, the user's previously determined affective state is initially set 
by defauh to "neutral". A loop commencing at 203 scans the keyboard for input. Upon 
obtaining keyboard input 204, the system proceeds to determine the user's apparent affective 

30 state 205, by way of means well known to those of ordinary skill in the art, as set forth by 



In any case, the present state of the art allows those of ordinary skill in the art, without 
undue experimentation or testing, to use face recognition software to analyze a user's facial 
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example above. Using, for example, the message content, input device force detection, 
autonomic indicators or facial expression, or any combination of methods well known to those of 
ordinary skill in the art to determine the user's apparent affective state at 205, the system 
compares the newly determined affective state to the previously determined affective state at 
5 206. If these states are the same, the system proceeds to output text at 210 without a change in 
formatting. 

Considering the initial condition of this aspect of the invention, the initial setting of text 
formatting by default corresponds with communicating a "neutral" affective state in this 
embodiment. Since the initial setting of the user's previously determined affective state is 
10 "neutral" at 202, if there is no difference between this initial state and the current affective state 
newly determined at 205, step 207 will cause the initial output of text that is appropriately 
f2 marked "neutral" by default at 210. If, on the other hand, the current affective state differs from 
the initial setting of the user's previously determined affective state ("neutral" in this 
ry embodiment) step 207 will direct step 208 to change text formatting appropriately and the initial 

;.. i.s 

r}j5 output of text will be appropriately marked at 210. 

'i: As set forth by example above, text formatting indicating emotional state may be 

- indicated 208 by way of highlighting the text in some way, such as alteration of font appearance 
y by color, underlining, bold, italic, capitals, etc. In the alternative, such marking may be by way 
L 3 of special character combinations or special characters inserted in appropriate textual locations. 
^Jo In any case, the information system will change text formatting at 208 in a manner well known in 

the art, marking the text in such a way as to be likely to convey the user's affective state to the 

intended recipient of the composition. 

Subsequent to the initial condition, if the current affective state determined at 205 differs 

from the previous affective state (which is to say the user's apparent emotional state has 
25 changed), the system changes text formatting appropriately at 208 as described above. At 209 

the variable indicating the user's present affective state is reset to indicate the current state as 

determined in step 205. Appropriately formatted text is output at 210. After this procedure has 

output text, it continues to scan the keyboard for input, monitoring the user's apparent affective 

state and modifying text markings as necessary and appropriate. 
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Referring next to FIG. 3, a flow chart is shown of a procedure for applying a 
determination of apparent affective state to prompting whether to transmit a user's electronic 
mail composition. This procedure, which is an application operating between the composition 
and the transmission of email, commences upon the user's activation 301 of a "send" function for 
5 email from the user directed to one or more recipients. The information system determines the 
user's apparent affective state at 302 by using, for example, the message content, input device 
force detection, autonomic indicators, facial expression, or any combination of such methods. In 
the alternative, or additionally, the information system may make its determination of the user's 
apparent affective state at 302 by facts surrounding the user's composition and transmission of 
10 text. As set forth above, such facts may include the period of time between a user's first access 
of an email message and the user's reply to that message, or the length of time used to compose 
the message related to the intended recipients of the message. In any case, the information 
system, by methods well known to those of ordinary skill in the art, makes a determination of the 
ry apparent affective state of the user at 302. If the apparent affective state of the user is such that 
rJJ the user would be well-advised to exercise particular prudence in transmission of email 303, the 
''U system will warn the user and prompt for confirmation at 304. FIG. 4 is illustrative of a typical 
warning and prompt, showing the circumstances leading the information system to determine 
LJ that the user's affective state might be questionable. If and only if the user confirms the prompt 
[ : 305, the email is transmitted at 306 and the program exits 307. 

^ Although the invention has been described with a certain degree of particularity, it should 

be recognized that elements thereof may be altered by persons skilled in the art without departing 
from the spirit and scope of the invention. Accordingly, the present invention is not intended to 
be limited to the specific forms set forth herein, but on the contrary, it is intended to cover such 
alternatives, modifications and equivalents as can be reasonably included within the scope of the 

25 invention. The invention is limited only by the following claims and their equivalents. 
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